Skip to main content

LSTM Architecture: The VIP Highway

In the last chapter, we saw how standard RNNs suffer from a terrible "telephone game" problem. Because they constantly multiply numbers at every step, their memory either vanishes into nothingness or explodes and breaks the computer.

To fix this, researchers in 1997 created a brilliant upgrade: the Long Short-Term Memory (LSTM) network.

The Two Roads​

A standard RNN only has one road for information to travel on: the Hidden State. We compared it to a blender. Every new word gets chopped up and blended into the hidden state. Over time, it turns into a muddy, unreadable mess.

An LSTM fixes this by building two separate roads through the network:

1. The Local Road (Hidden State)​

This is exactly like the old RNN. It's a short-term memory that holds the messy, blended-up details of the current word.

2. The VIP Highway (Cell State)​

This is the LSTM's superpower. The Cell State is a fast, protected highway that runs straight through the entire network, from the first word to the last.

The Secret to LSTMs: Information on the VIP Highway (Cell State) doesn't get violently multiplied and blended at every step. Instead, the network gently adds or removes information.

Because it relies mostly on addition instead of multiplication, the error signal can zoom backward down the VIP Highway at light speed without vanishing or exploding!

The Toll Booths (Gates)​

If the VIP Highway is so fast, how do we prevent it from getting clogged with useless junk?

The LSTM places Toll Booths (called Gates) along the highway. As a word moves through the network, the AI acts like a traffic cop. It looks at the current word and the short-term memory (Local Road), and makes a decision:

  • "Do I need to erase old information from the VIP Highway?"
  • "Do I need to add this new word to the VIP Highway?"
  • "What should I show to the outside world right now?"

These traffic cops are what allow an LSTM to remember things for hundreds of steps! If it sees the word "France" at the beginning of a paragraph, it can put "France" on the VIP Highway and protect it all the way to the end, perfectly translating the language.

Next Up: Let's zoom in on these traffic cops. How exactly does the math inside an LSTM's "Gates" work?